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Abstract. In biological systems, information is frequently transferred with Poisson like spike processes 
(shot noise) modulated in time by information-carrying signals. How then to quantify information transfer 
for the output for such nonstationary input signals of finite duration? Is there some minimal length of the 
input signal duration versus its strength? Can such signals be better detected when immersed in noise 
stemming from the surroundings by increasing the stochastic intensity? These are some basic questions 
which we attempt to address within an analytical theory based on the Kullback-Leibler information concept 
applied to random processes. 



PACS. 05.40.-a Fluctuation phenomena, random processes, noise, and Brownian motion 
alytical theories - 87.10.Vg Biological information 
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1 Introduction 

Stochastic resonance (SR) [1] grew into extensive research 
domain on the border between many scientific disciplines, 
ranging from geophysics and climate dynamics, numer- 
ous physical, biophysical and engineering applications [21 
3 , 4 , including quantum SR in the deep quantum cold [5] . 
Nevertheless, more complex physical SR applications and 
timely SR applications to biological and climate complex- 
ity [6] as well as more insightful reasoning are still in the 
limelight. The original statement of the problem, i.e. a 
paradoxical amplification of the signal in a noisy back- 
ground due to intrinsic [7], or added, external noise [2 
3,4 , has been contrasted with a synchronization frame- 
work [2,3,4,8]. Post] "actum we can reformulate the origi- 
nal problem by asking the question of whether a stochas- 
tic bistable clock can resonate with an externally applied 
periodic driving via increasing the randomness of the un- 
derlying bistable clock dynamics. 

The notion of a "Stochastic clock" [9] stems concep- 
tually from the theory of continuous time random walk 
processes [see e.g. in Ref. [El, P- 245]. It is characterized 
by a distribution of the sum of stochastic periods. The 
distribution of one period duration is a convolution of the 
residence times in the two clock states. Two subsequent 
transitions perform a cycle with a random duration. It is 
important to note that if the mean duration of a cycle 
{Tcycie} exists, the distribution of n cycles duration yields 
a sharp function, centered at n(T cyc i e ) in the limit n — > oo. 
Intrinsic noise changes (r cyc / e ) and in some situations, - 
e.g. for symmetric Markovian clock with an exponential 
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distribution of the residence times and an exponential de- 
pendence of the mean cycle durations on the noise in- 
tensity -, the stochastic clock can resonate with a weak 
periodic driving of period Tq, when (r CJ , c / e ) = Tq. This is 
the benchmark of the stochastic resonance phenomenon. 
Then, the periodic signal is best detectable in the spectral 
power spectrum of the clock's bistable fluctuations and 
the stochastic transitions become more correlated with the 
periodic time course of the signal. 

Whether the information transfer will be optimized at 
this resonance condition depends on how is information 
encoded. If a direct encoding is used, i.e. locally in the time 
domain, then the answer is "yes" . However, if information 
is encoded in the frequency domain (like used in radio 
devices), then for the discussed bistable clock the answer 
is typically "no", at least for weak signals. The spectral 
signal-to-noise ratio (SNR) characterizes the Shannon in- 
formation transfer for weak signals [11] (more precisely, 
the information capacity of an information channel which 
is the maximal rate of the Shannon's mutual informa- 
tion between the input and output signals for the fixed 
total power of the input signal). Stochastic resonance in 
the spectral SNR for the stochastic bistable system does 
not necessarily reflect a synchronization phenomenon J5J 
I3HMH] • More specifically, SNR does not directly reflect the 
matching between the stochastic time scale of the bistable 
clock dynamics and the time-scale of the deterministic, 
coherent signal dynamics. In contrast to SNR, the mea- 
sure of spectral amplification 12,13], however, explicitly 
involves a dependence on the driving frequency. We also 
remark while within linear response (i.e. weak signals) the 
SR gain determined by the "SNR output/ SNR input" 
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Fig. 1. (Color online) A step-like signal modulates an 
information-carrying spiking process, e.g. output of a sensory 
neuron |2"7] . 



cannot exceed unity [14] this is no longer the case for the 
nonlinear SR response [T4TI15] . 

Next we are dealing with SR in a wider sense, i.e. we 
shall study SR in a form which can broadly be character- 
ized as a relative amplification of the information transfer 
through a noisy system. SNR for a weak sinusoidal sig- 
nal predicts the information transfer for stationary, weak 
broadband stochastic signals pn[TBlH~7] . As just noted, 
this remarkable analogy fails, however, for strong signals 
beyond the linear response approximation 14,15j. Never- 
theless, it must be remarked that it is simply not feasible 
to extract more information (in the Shannon sense) from 
the output signal than was originally encoded in the input 
signal; this agrees with common sense and is corroborated 
with the information processing inequality [11) . 

In the case of so-termed aperiodic SR, i.e. SR fed by 
stationary stochastic input signals (modeled, e.g., by Gaus- 
sian processes) the rate of mutual information can be used 
as a suitable quantifier [18j . How to proceed, however, if 
the signal is not stationary as it is intrinsically the case 
for fixed, deterministic or stochastic non-stationary inputs 
of finite duration? This latter situation is typical, e.g., for 
biological systems [HI3JH], cf. a typical situation depicted 
with Fig. [T] Typical spectral measures as indicated above 
are then of limited use, or at best of approximate use only. 
The mutual information concept also cannot be applied 
whenever the input signal is strictly deterministic. In such 
a nonstationary situation one can characterize the infor- 
mation transfer by the change of the entropy between the 
process in absence of input and the output process when 
the input signal is applied. Then the difference of entropies 
can be regarded as the information gained from the input 
signal [11]. The Kullback-Leibler relative entropy [19,20, 
I21j , termed also the information gain is a suitable measure 
to characterize the corresponding entropy difference be- 
cause it does not suffer from the subjective dependence on 
the discrete time-step At used in approximating contin- 
uous time random processes [22II23] . This presents an ad- 
vantageous fact when contrasted with using direct entropy 
differences (T5J- The Kullback-Leibler entropy is just an 
analogue of the Boltzmann H— function for kinetic equa- 
tions [23]. In the present context, it is applied not just to 
a single time probability density, but rather to the whole 
probability functional that determines the stochastic pro- 
cess under consideration. Like the H— function it charac- 
terizes the entropy difference from equilibrium in a well- 
defined manner, even if the equilibrium value of informa- 



tional entropy itself is not precisely defined for continuous 
distributions. This is always the case if some fundamental 
"quantum-scale" (like the Planck constant for an elemen- 
tary "area" in the phase space of a physical system, in the 
case of physical entropy) is absent. Moreover, the use of 
information gain if averaged over all possible realizations 
of a random input signal provides an upper bound for the 
mutual information transferred [22U23j . These facts pre- 
destine indeed the information gain as an adequate mea- 
sure to characterize nonstationary SR. 

With this present work, we investigate such nonsta- 
tionary SR for an archetype setup of SR (2[[23,25,26], by 
considering the renewal point processes for signal-modulated 
spike occurrences. This sort of modeling is relevant to sig- 
naling occurring in biological systems [16,27,28,29]. 

2 The model 

We consider a renewal point process £(t) defined by the 
spikes 

^t) = ^S(t-U) (1) 

i 

occurring at random times tj, see in Fig. [5] One may safely 
assume that the spike width is negligible and its form 
is fixed by some total intensity (the time integral of the 
spike-form) £, i.e. the information is transferred with the 
spike occurrences. Put differently, rather than a specific 
shape of the spike it is the timing dynamics and not only 
the averaged number that is important in transferring in- 
formation. The interspike intervals (ISI) tv = — U are 
assumed to be uncorrelated (renewal assumption) and the 
whole process, generally a non-Markovian process, is com- 
pletely characterized by the ISI distribution ip(t + r,t) := 
^(r\t), or equivalently by the corresponding survival prob- 
ability <P(t + r,t) := <P(r\t), il){r\t) = -d$(r\t)/dT (30j 
131] . The process is non-homogeneous in time what is re- 
flected by its explicit dependence of the above quantities 
on the current time t via an input signal V s (t). For a time- 
homogeneous process we have in contrast ^){r\t) = V'(' r )- 
The simplest example is given by the Poisson process with 
the time-dependent rate r(t) [24ll34j . where 

*(t|*) = exp f- J + r(t')dA . (2) 

A popular SR model is [2 ] I26 P 25]: 

r(t) = r (U )exp(qV s (t)/D) , (3) 

where 

r (U ) = k eM-Uo/D) (4) 

is the rate in the absence of signal. One further assumes 
that there is a threshold Uq which upon crossing induces 
a spike. Typical realizations are a conventional threshold 
detector [35] , or the dynamics of the leaky integrate-and- 
fire model of neuron firing [27] driven, e.g., by synaptic 
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Fig. 2. Driven renewal process: spikes occur at random times 
ti, the interspike time-intervals n = ti+i — ti axe assumed to be 
uncorrelated and described by a non- homogeneous ISI density 
V>(r|i). 



noise. Some intrinsic noise of the intensity D produces 
spontaneous firing and the signal modulates the threshold 
height. A similar model applies also to the current spikes 
produced in a membrane by spontaneous electroporation 
facilitated by some ion channel forming peptides |26j . In 
the latter case the signal V s (t) is the voltage modulation, 
the quantity q is an effective gating charge, Uq is the en- 
ergy barrier to the channel formation, and D = ksT. A 
similar model (modified for the refractory times) provides 
also a crude approximation to the activity of cortical and 
sensory neurons [37] , spontaneous spiking of the ion chan- 
nel clusters [36], and spontaneous calcium release spiking 
in living cells [29"] . 



3 Theory 

We start out from the trajectory description of £(t) con- 
sidering a finite time interval [£o, £)- The elements of the 
probability space are the trajectories with spikes occurring 
at some random times ti, tsj ■••!*«• The probability to have 
spike occurring during the prescribed interval (starting 
out from a no spike event at time to ) is Po(t, to) = $(t, to). 
The probability density of trajectories with one spike oc- 
curring at ti, to < ti < t is 

Qi(t,h,t ) =#(f,ti)V(*i,to), 
yielding for the corresponding probability 



Pi{t,t ) = / Qi(t,ti,to)dti . 

Jto 

Furthermore, the probability density of trajectories with 
two events at t\ and tg, to < ti < ti < t, reads 

Qa(t,*2,*i,to) = *(i,t 2 )V(t2.*i)^(*i.*o). 

and the probability to have two events within [to,f) be- 
comes 

ft(*,*d)= / dt 2 f tftiQ a (*,*a,*i,*d). 



thus obtain a complete description of the considered time- 
inhomogeneous, nonstationary process with the probabil- 
ity density functional 

tn : • ■ ■ j tl , to) , . . .J . 

For this rate-modulated Poisson process the densities read 



) n (t,t n , ...,ti,t ) = exp 



f r{t')dAj[r{U), (5) 



and the number of spikes exhibits a Poisson distribution 
P n (tM)= (n( ^ 0)) " exp[-(n(t,t ))] (6) 
with mean (n(t, to)) = J. r(t')dt' . 



3.1 t— Entropy 

The definition of the entropy of any continuous variable 
which has a physical dimension requires to introduce some 
arbitrary bin At (a measurement unit). The entropy of 
£(t) can be defined (in natural units, nats) as a functional 
integral [37J 

S A r(t,t Q ) = -P (t,t )lnPo(t,to) 

00 pt ft-n, fi2 

~J2 dtn / dt n-i- / dh (7) 
Q n (t,t n , ...,t )ln[Q n (t,t n , ...,t )(AT) n ] . 

Upon differentiating Eq. ([TJ with respect to time t we 
arrive after some algebra at the following expression for 
the rate of entropy production 

d r* 

S A r(t, to) = F{t, t ) + / F(t, t!p{t u to)dh (8) 



df 



in 



00 pt ptn P?2 

= F(t,t ) + ^2 dtn\ dt^... dtl 



l \ u F(t,t n ) 



xQ n (t, t ni to) 
In Eq. jg]), 



F(t,tx) := #(t,ti)ln 



-Ar#(i,*i) 



(9) 



where <£(t,ti) = d<P(t,ti)/dt < and 

00 ft ftn 



<f(t,t ) = 2 / dt„ / dt n - x ... I di x 



Other probability densities and probabilities are constructed 
akin, using the semi-Markov, renewal character of the un- 
derlying process. The normalization condition 

S^Lo Pn(t, to) = 1 can be readily verified; it is done by Using that 
showing that the derivative of the l.h.s. with respect to 
t is zero upon using Po(*0>*o) = U -Fn^o(*0)*o) — 0. We 



n^.ti-i). (io) 



i=l 



<P(0\t) = 1, lim <P(r\t) = 0, 
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one can show that S/ji-C^I*) = f °° F(t + T,t)dr has the 
meaning of the entropy of the time-inhomogeneous ISI, 
i.e. 



SAr(ip\t) = - ip(r\t) \n[i>{T\t)AT]dT . (11) 
Jo 

For the Poisson model, &(t,t Q ) = r(t) and Eq. © 
simplifies to 

J t S ^{tM)=r{t)\n(^^j =r(t)S AT (m, (12) 

where Sat (ip\t) is the entropy of the ISI distribution which 
is calculated with the frozen rate r(t), i.e. with ip( T ) = 
r{t) cxp(— r{t)r) instead of ip( T \t) m Eq. (fTTjl . This result 
has a simple interpretation: namely that the rate of en- 
tropy production — spiking rate x entropy of ISI distribu- 
tion for instant rate r(t). For the "background" process, 
i.e. the resulting process with no signal applied, the rate 
is ro and the entropy of the spike train of duration T is 
given by the well-known MacKay and McCulloch result 

MM 



So = N(T) In 



tqAt 



(13) 



where N(T) = r^T is the averaged number of spikes. A 
popular definition of the information Iat(F) transferred 
with spikes [16l[T7] amounts to take the difference 



Iat{T) — S - Sat^o + T, t ) 

fto+T r 



(14) 



r ln 



r Ar 



r(t) In 



r(t)Ar 



dt. 



One can see that the dependence on the finite time bin At 
does generally not cancel [23*pS0] , unlike in the case of an 
n-dimensional probability distribution. The reason is that 
the probability P n to have n spikes is changed, i.e. prob- 
ability is redistributed between different n-dimensional 
"slices" of the hyper-dimensional probability space. Trou- 
blesome is further the finding that the above difference 
can readily become negative (i.e. for r(t) > ro and for 
a sufficiently small At). This should then be interpreted 
as a loss of information. Of course, a proper definition 
should always yield a positive information, because the 
spikes become more ordered due to the application of the 
input signal. The interpretation problem is an artefact of 
this At dependence. This is precisely why we prefer to 
define the entropy difference for stochastic processes via 
the Kullback-Leibler relative entropy, see below. Before 
we proceed with a suitable definition along our posed ob- 
jective we derive next a generalization of the MacKay and 
McCulloch result for different, nondriven (i.e. V s (t) = 0) 
point processes. 

3.2 Rate of entropy production for stationary renewal 
point processes 

Using the exact result © one can find also the asymp- 
totic rate of entropy production for time-homogeneous 



processes in the limit t — > oo for any ip( T ) with a finite 
mean ISI (r) = j Q Tip(T)dT. In this case F(t + T,t) — 
F(t), )P(t + T, t) — ^(t), and for the Laplace-transformed 
rate of the entropy production Sat (i, 0) we obtain 



R(s) 



F(s) 



(15) 



1 - 1>{8) 

The asymptotic rate of entropy production follows as 

Um ^-S A r(t,0) - = t^SatWO > (16) 

t^oo at s— >0 (T> 

which is a natural generalization of the relation (fT!?|) . The 
role of the mean spiking rate is taken on by 1/(t). The 
result in (fT3"]) is thus generalized to read 



So = N(T)SatW, 



(17) 



where N(T) = T / (r). This latter result is applicable also 
in the case of fractal-rate renewal processes [39] , where the 
mean rate does not exist, i.e. where ip(s) ~ 1 — (sr*) Q , for 
s — >0;0<a<l and r* is some scaling time parame- 
ter [ID] , The only difference is that the number of spikes 
within a long time interval T scales sub-linearly with its 
length, i.e. N(T) oc (T/r*) Q . Generally, for the consid- 
ered renewal processes we find that entropy of spike train 
= number of spikes x entropy of ISI. One can also infer 
that the Poisson process is the maximum entropy point 
process, for fixed (t) and At. This is so, because the ex- 
ponential ISI distribution displays the maximum entropy 
distribution from all one-sided distributions under such 
constraints. 



3.3 Kullback-Leibler relative entropy 

As discussed above, r-entropy of the spike train is not 
exactly defined, being dependent on At. However, its de- 
viation from equilibrium can be defined unambiguously 
via the relative entropy, given by the functional integral: 



#[to,t]K(*)l6>(*)] 



D[at)]pm)\Vs(t)]in 



, PMt)} 



00 ft fin P^2 

^ / dt n dt n _ 1 ... / dtiQ n (t,t n , ...,t \V s ) 



\ Qn {t, t n: ...,to) ) 



(18) 



Here, the super-index (0) refers to the background process 
£ {t) in the absence of signal V s (t). Relative entropy is al- 
ways non-negative. It is zero iff the both distribution func- 
tionals P[£o(*)] and P[£(t)\V s (t)] coincide i.e. in the 
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Fig. 3. Kullback-Leibler entropy per one spike as a function 
of the frozen rate r(t). 



absence of signal. The rate of the relative entropy produc- 
tion for the rate-modulated Poisson process can be easily 
calculated with Eq. © in {T5J) [23]. It reads, 



dt 



K lt0 , t] m)Mt)} = r(t)ff*(V#o) > 0, (19) 



where 



WW= / dr^(r)ln 
Jo 

~ r(t) 



1 + ln 



-0(0) ( r ) 
ro 



(20) 



with the time-dependent transition rates ri^(t) detailed 
e.g. for SR gating in ion channels in Ref. [22]. Using the 
present notation the rate of information gain then emerges 
as 



dt 



K [to<t] [x(t)\x (t)} = n(t)K t (^\4 0) ) Pl (t) , (22) 

»=1,2 



where the relative entropy is given again by Eq. (|20[) for 
the residence time distributions in the two states, read- 
ing -0i ; 2 (t) = f*i,2(t) exp[—r"i ,2 (t)T] with the frozen rates 



n j2 (i), and 



(0) 
1,2' 



exp(— rf'lr) are the rates in 



is the Kullback-Leibler entropy of the ISI distribution with 
the frozen rate r(t) with respect to the unperturbed ISI 
distribution with the rate r . The meaning of the result 
in Eq. (| 19() is as follows: the rate of information transduc- 
tion = spiking rate x relative change of the ISI entropy. 
Notably it does not depend of the time bin At. The to- 
tal information about signal is the time-integral of Eq. 
(fT9|) . Fig. [3J depicts the relative entropy for the exponen- 
tial distribution as a function of r(t)/ro- Notice that the 
information transferred per one spike can in principle ex- 
ceed one bit for a strong rate-modulation, in accordance 
with [16 . This is because the information is transferred 
not only via the spike occurrence, but also with its timing 
(and time is a continuous variable). For the weak signals 
(qV s < D), the information transfer per one spike, how- 
ever, does not exceed one bit, independently of the time 
resolution At. 



3.4 Rate of information gain for a two-state process 

A generalization of the above result for the two-state Marko- 
vian rate-modulated process x(i) can also be obtained [22 . 
The Markovian two state dynamics is governed by the 
master equation 

Pl (t) = r 2 {t)p 2 {t) - n{t) Pl {t), P2 (t) = 1- Pl (t), (21) 



the absence of the signal. In [22] this result has been ap- 
plied to investigate the problem of stochastic resonance 
in biological ion channels [26j from an information theory 
perspective. 



4 Nonstationary SR 

Now we are sufficiently equipped in order to address the 
questions posed in the abstract. We adhere here to the 
Poisson model and apply a transient step-like signal of 
the amplitude A and the duration To depicted with Fig. 
[I] i.e. V a (t) = A for t < t. m < t < t ln + r , V s {t) = 
otherwise, and where tj„ is the time instant when the 
signal is applied. The signal can be either positive, A > 
(activating signal), or negative, A < (inhibiting signal). 
In terms of the averaged number of background spikes, 
No = ro To, occurring within a typical time interval of the 
duration To in the absence of signal, the total information 
gain is evaluated to read 

K = tf [t0 ,oo][£(*)M«)] = N [r(A)/r ]K(ip\ip ), (23) 

where r(A) = r(t) = const for U n < t < U n + tq and 
K{tp\ij) ) is given by Eq. (20]) with r(t) = r(A). This is 
result after integrating Eq. (|19p . Its structure is illumi- 
nating; in terms of Nq the signaling information transfer 
involves both the change of the spiking rate with applied 
signal r(A) and the relative change of the ISI-entropy. 
Interestingly, for r(A)/ro < 1 (inhibiting signal) the in- 
formation transferred per one background spike cannot 
exceed one nat, cf. Fig. [D Put differently, then each dis- 
appeared background spike bears no more than one nat of 
information, in accord with intuition. 

Furthermore, for the popular model in Eqs. ©, 
we obtain a practical result, reading 



K = fcor exp ( - — 



.(24) 



For very weak signals, q\A\ <C D, it displays the well- 
known, bell-shaped SR dependence on the noise intensity 



K 



, (qA) 2 
koto „ 2 exp 



Uo 
D 



(25) 



being proportional to the SNR in the case of sinusoidal 
signal with the same amplitude [21 1231125] . The latter re- 
sult confirms the fact that the use of SNR indeed describes 
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r(A)/r 

Fig. 4. Information gain K per one spike of the undriven Pois- 
son process (in nats) as function of the relative change of the 
spiking rate r(A)/ro. 



information transfer for weak signals. In order to detect a 
signal, the total information gain should intuitively be no 
less than one bit. This clearly poses a bound on the signal 
duration To depending on its amplitude strength A. This 
bound can be found from Eq. (|24|) : Weak signals should 
last for a sufficiently long time To (many affected spikes ac- 
cumulate the corresponding information), otherwise such 
signals cannot be detected within environmental noise. In 
Fig. we depict the information gain K versus the noise 
strength D for several values of the signal strength A. The 
threshold barrier Uo is set to Uq = 75 meV. Assuming that 
q is equal to the elementary charge, the signal strength is 
measured in mV. 

The presence of wide sense SR is clearly detectable in 
Fig.[DJa-c), for both activating and inhibiting signals. For 
an activating signal of the threshold strength, qA = Uq, 
in Fig. [SJd), nonstationary SR disappears. The increase 
of the information transfer by increasing the randomness 
of the background process has an instructive explanation. 
Namely, the increase of the spontaneous spiking rate leads 
to more spikes occurring within the signal duration To- 
They altogether transfer more information about the sig- 
nal. From the value of K in Fig. E^a) one can realize that 
many spikes are required in order to transfer information 
of K = 1 nat (approximately 0.7 bit) about the corre- 
sponding weak signal. Namely, one needs /co T o > 2 ■ 10 4 . 
This clearly poses a bound tq > 2 - 10 4 /ko on its duration. 
Assuming that ko = 20 ms -1 (such that tq « 1 ms _1 at 
D = 25 meV), this yields the bound tq > 1 sec, i.e. such 
a faint signal should last at least for about 1000 spikes to 
become detectable. However, if to increase the strength of 
the signal to 10 mV, the corresponding bound for tq drops 
by two orders of magnitude, as it can be deduced from Fig. 
[5]Jb). In other words, such a stronger input signal (being, 
however, still much below the threshold) can be detected 
already with a few spikes, in principle. Moreover, nonsta- 
tionary SR can help to detect such signals which would 
otherwise far too short lived at a non-optimal noise inten- 
sity D. 



5 Conclusion 

In this work, we considered a basic model for nonstation- 
ary SR, i.e. for the case of deterministic but aperiodic 
signals of finite duration. Information theory helped us to 
shed light on the very possibility and the origin of such 
nonstationary SR, as well as other critical issues such as 
the existence of a bound on the signal duration To versus 
its strength A. The obtained results may be of a broad 
importance in the context of information transduction in 
biological systems on the cellular level and in sensory sys- 
tems. 
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